docs(agents): require an oracle, not a model of one, for deciding predicates - #300
docs(agents): require an oracle, not a model of one, for deciding predicates#300vsits-proxy-builder[bot] wants to merge 2 commits into
Conversation
…dicates PR #283's ca-trust guard merged with two approvals and independent verification of every blocker, and is wrong in both directions on main — a CRL in a bundle voids the whole file, and our CA relabelled TRUSTED CERTIFICATE passes a check node's loader would fail. Neither reviewer ever fed the guard a realistic bundle. The suite was green because it exercised a hand-copied twin: mutating the shipped guard to accept unconditionally left it at 12/12. Adds the rule and its two corollaries — the oracle must be the same API production calls, and a test must be proven to reach the shipped code before it counts as evidence — plus the phrasing rule for universal claims like "conservative, never permissive". Widened past trust decisions to any predicate that predicts another program, per AI Team Lead: git push --dry-run against a branch ruleset is the same failure with a different oracle.
|
Reproduced both defects on Both claims reproduceNode v24.11.1.
False accept and false reject, as written. The 38 ms is the spawn floor, not the oracleThe scope note reads "an oracle is not free" off a 38 ms median for a bare Once per launcher start, not per request. And the launcher already forks node at What the oracle buys, against ground truthGround truth here is not a second opinion — it is handing the bundle to a child via Against the real 132-cert bundle on this host (publishers: our CA, a third-party pin CA, system roots):
Predicate 2/3 wrong; oracle 3/3 right. Those two shapes are the open blockers on #296, which is round 6 of the five this PR counts. The generic argument, and a limit on my own evidenceThe loader does not know who published a certificate. An oracle reads what it actually loaded and looks for the CA you asked about, so the same code path answers for any publisher with a different needle. A predicate parses file syntax, so every additional publisher is more shapes it has not seen — a third publisher arriving on this host is that risk increasing, not a hypothetical. Limit, stated because the rule this PR adds is about exactly this: my harness serves our own leaf, so the rows I ran for the third-party CA prove nothing about its trust path. The handshake passed on our CA inside the bundle. The publisher-agnostic claim above is structural plus matching verdicts, not an end-to-end handshake for that publisher. One more instance of the same class, upstream of the guardThe bundle builder has the same shape. It validates the concatenation with an awk A bundle the builder certifies as healthy, from which node loads nothing. Worth naming in "Where else it applies": the count is a model of the parser, and the parser is one spawn away. On the rejecting directionThe PR is right that rejection is not safe here, and today's fallback is worse than the write-up suggests: a refused bundle means the launcher uses only its own CA, so a third publisher's CA is dropped for that session. Measured — Not asking this PR to decideThe design question is parked with Chris and me, and touches two other implementations, so this is evidence toward that decision rather than a request to change this PR's scope. The rule and both corollaries land as written regardless. My reading: the increment is 4 ms once per launch, the predicate is measurably wrong in both directions on a real bundle today, and the round count is the argument the PR itself makes. — codeslake (CCF contributor) |
|
Implemented the oracle in #296 and it found three more instances of this rule's own failure mode — two of them in code I wrote while applying the rule. Posting them because they sharpen where the corollaries need to bite. The rule caught a wrong row in our own test tableRunning the existing 26-shape table against the real loader, one row disagreed:
That row recorded the predicate's behaviour as the expectation. Five review rounds, three parties, and it survived every one — because each round compared the code to the table and none compared the table to node. Corollary B says verify a test reaches the shipped code; this is the sibling failure: the test reached the code fine and asserted the wrong answer. A green suite is not evidence, measured twice moreThe probe API did not exist on half the runtimes we support. The launcher crashed on a path 43 green tests never touched. Every test built its CA with Neither TDD nor an intermediate-value check caught that one. What caught it was running the real entry point. That is a third corollary and I think it belongs in the section:
On the cost figureThe 38 ms in "Not in scope here" is the bare-spawn floor, not the oracle. Measured, 100 interleaved runs so both share load conditions: Once per launcher start, on a path that already forks node for the proxy ( Where else it applies — one more instance, upstream of the guardThe bundle builder has the same shape one level up. It validates the merge with an awk A bundle the builder itself certifies as healthy, from which node loads nothing. And What the oracle does not doWorth stating so nobody deletes the builder's check believing this replaces it: the launcher's question is self-carry. It holds one CA and can only ask about that one, so a merge that silently dropped a sibling publisher answers Cross-version, post-implementation: full suite 1512/1512 on Linux node v24.11.1; 1511/1511 with one linux-only skip on macOS node v26.5.1 and v25.8.0. — codeslake (CCF contributor) |
Five rounds on the CA guard argued node CA-loader semantics. The client stopped being node at CC v2.1.113 — documented in AGENTS.md, README.md, and CHANGELOG.md, and the reason the NODE_OPTIONS preload died and this proxy exists. Every round had it available; none consulted it. Also adds the expectations rule: mutating the code to prove a test reaches it is not enough when the test asserts what another program does — the expected value must have come from that program. A shape-table row recorded the predicate's own behaviour as the expectation and five green rounds re-certified it.
What this fixes
We approved a broken trust guard. PR #283's
ca-trustcheck merged as23346ac9with two Codex rounds, independent verification of every blocker, and a green suite — and it is wrong in both directions onmaintoday:new X509Certificate(block)runs on every PEM block and the throw escapes to the outercatch. One CRL in a bundle voids the whole file. Rejection is not the safe direction here: the fallback drops every sibling and corporate CA for the session, which is the exact failure the contract exists to prevent.X509Certificateignores the PEM label, so our CA relabelledTRUSTED CERTIFICATEyields byte-identical DER and passescarriesUs, while node's loader skips any block not labelled exactlyCERTIFICATE.Both reproduce on
origin/main. Both were reproduced independently by the AI Team Lead before this rule was agreed.The review process worked on everything it was pointed at — a write→rename race measured at 0.88 ms over 5,000 iterations, the rendezvous path grepped, file modes checked. It was pointed at the wrong thing. Nobody ever fed the guard a realistic bundle, on the function that decides what the client trusts for the host its API keys travel to.
The corollaries are the point
The headline rule alone would not have caught this, which is why both corollaries are in the section rather than as asides:
tls.connect({ca}), while the launcher usesNODE_EXTRA_CA_CERTS. The two disagree. Green test, certified nothing.test/proxy-forward-ca.test.mjspassing 12/12, because the test exercised a hand-copied twin. "We have tests" was counted as reassurance by two reviewers.Scope, per AITL
The rule fires on any predicate whose job is to predict another program's behavior, not only on trust decisions. The unifying property is that the oracle exists and we chose to model it instead of calling it.
git push --dry-runas a test of a branch ruleset is the same failure — it reports success against a ruleset the server never consulted; we hit that one this week too.Not in scope here
The design question — why a predicate rather than an oracle at all? — is open with Chris and @codeslake. AITL measured a 38 ms median floor for a bare
nodespawn on this host, so the honest version is that an oracle is not free. That decision also touches two other language implementations of the same contract and isn't made in this PR.Non-Functional Requirements
## Evidence Classand## Anti-Bloat Lens, the two sections it composes with. No new abstraction.Ref #293
— Proxy Builder